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ABSTRACT 

The current study aims to contribute to classroom instruction of undergraduate education faculty students through development of an online contest, and 
evaluation of test items based on participants’ responses. It is a pilot study which covers four units of a single course offered in the Department of Computer 
Education and Instructional Technologies at Anadolu University. Students are provided with the opportunity to practice their skills on-line whenever and 
wherever they want. The research process involved four successive and interdependent steps, namely design, development, application and evaluation. An 
on-line testing program which has both a learner and instructor version was prepared. Questions were gathered from different sources along with the ones 
prepared by researchers. Item facility, item discrimination and distractor efficiency indices of questions were calculated by the program after it was piloted 
with 32 undergraduate students. Features of the program and implications for instruction were provided as well. 
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INTRODUCTION 

Assessment is an important factor in the success of university education (Phillips & Lowe, 2003). Despite widespread use of 
computers in teaching and learning endeavors, their use for assessment purposes seem to be limited. Computer adapted 
testing has been used with hesitation maybe because of its novelty or its limited feedback ability on authentic instructional 
tasks. In recent years, availability of course management software like Blackboard and WebCT made Internet increasingly 
attractive for assessment purposes. Computer based assessment not only facilitates routine tasks like grading multiple-choice 
tests, but also can empower students’ learning experiences (Brown, Race & Bull, 1999). Using computers in assessment has 
several benefits as suggested by Brown et al. (1999), Cann and Pawley (1999), Danson (1999), and Harvey and Mogey 
(1999). These benefits can be listed as follows: 

Large number of papers is marked quickly which reduces the load on teachers. 

Immediate feedback is given. 

Responses are coded without coding errors. 

Responses are directly transferred to statistical software packages, which facilitates further item analyses. 

Responses of participants can easily be monitored and classified. 

Assessment can be stored and reused. 

Assessment items can be randomly selected to provide different test sheets for each student. 

On the other hand, as suggested by Pain and Le Heron (2003) creating a question database for computer adaptive assessment 
is time consuming. Test delivery may differ by the type of browser. The infrastructure might be insufficient to administer on- 
line assessment as observed in a recent study (Harwood, 2005). Thus, on-line tests should be prepared with caution so that it 
is applicable in as a large variety of computer settings as possible. 

A distinction between formative and summative assessment is usually made while examining the purpose of the assessment 
(Biggs, 1999). Formative assessment provides feedback to students to help their learning while summative assessment is 
generally used to grade students at the end of a course. Bransford, Brown and Cocking (1999) review studies on 
developmental and cognitive psychology and maintain that formative assessment is particularly important since it improves 
the quality of thinking and understanding, provides regular feedback, and creates opportunities for revision. The current 
paper provides an online formative assessment tool where self-assessment is realized by participating students. 

Harvey and Mogey (1999) summarize the problems related to computer adapted testing and suggest that practitioners should 
not be too ambitious while realizing a computer-based assessment process. Rather, they should start with small but 
manageable project. Thus, the study starts with small segments of an information technology course offered at a computer 
education and instructional technologies department in Turkey, and builds on the results to develop the project further. As 
suggested by Dowsing (1999), there should be harmony between the technological nature of the subject matter and some of 
learning and assessment practices. Students of BTO 101 - Information Technology in Education I course are supposed to be 
proficient at using technology. Thus, they constitute an appropriate sample for online computer adaptive assessment 
practices. 



METHODS AND PROCEDURES 



Participants 

Thirty-two undergraduate students enrolled at the Department of Computer Education and Instructional Technologies who 
took Information Technology in Education I course at Anadolu University participated in the current study. They produced 
responses to items which were further developed through the analyses conducted by the program. 

Software: 

Four units of BTO 101 -Information Technology in Education-I course were included in the contents. The units were 
Information System and Introduction to Computers, Computer Hardware, MS-Word and MS-Excel. After a table of 
specifications was prepared, some questions were prepared by researchers while others were adapted from several sources 
used during the course such as Onder, Qakir and Goksel (2000), Rua and Oztiirk (1995), Saatgi (1993). 
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Figure 1 : Reader version: Introductory page 



While preparing the software program, PHP and MYSQL were used. A reader and an author version of the program were 
published online. Entrance to the program was realized through providing a user name and password (Figure 1). 

When students enter their usernames and passwords, they confront with the rules of the contest. On the left side a menu is 
provided where students can update their membership information, see their grades along with the number of their correct 
and incorrect answers, their place in the whole group, and the list of all members along with their grades. Figure 2 provides 
the screen where students are able to see the list of all members and their grades. 
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Figure 2: Reader version: List of the whole group 

Students are allowed to enter the page as much as they want. They are allowed to exit and restart the contest any time they 
wish. On all pages, students have access to a link where they can start or resume the contest. Questions are randomly selected 






by the program from a 200-item question pool. Twenty seconds are allotted for each question during which students are 
supposed to select from four distractors. Whenever a student cannot get the answer right or cannot find the answer in 20 
seconds, that question is randomly asked again in subsequent pages. A sample question is provided in Figure 3. 
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Figure 3: Reader version: A sample question from the contest 

Beside the reader version, the program has an author version where instructors can add new questions or delete inappropriate 
questions with bad item facility, item discrimination and distractor efficiency indices. The page to enter new questions is 
provided in Figure 4 below: 
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Figure 4: Author version: The page for entering new questions 

In order to examine the contribution that each item is making to the test, item analysis is realized by the program as suggested 
by Hughes (2003). Item facility (IF) values are calculated by the program for each item. Item facility is the proportion of 
students who answered a particular item correctly (Brown, 1996). Thus, if 9 students out of 10 answered an item correctly, 
the item facility value is 90 %, which means that the item is very easy. Items are also examined in terms of their item 
discrimination (ID) index values. Item discrimination is the difference between the item facility values of the high achievers 
(usually the top 33 %) and the low achievers (usually the bottom 33 %) in a class (Brown, 1996). If this difference is high, it 
means that the item can efficiently differentiate between students who know the answer from those who do not. Finally, the 
distractor efficiency analysis (DEI) was utilized to eliminate the distractors that were never or rarely preferred by the pilot 
group. Besides, tricky distractors chosen by high ability learners and ignored by low ability learners can also be eliminated by 
the program. These analyses are particularly applied to improve the reliability of the test, which are calculated and displayed 
by the program within a single screen (Figure 5). 
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Figure 5.: Author version: Analysis of questions 

CONCLUSION 

The current study suggests an online formative assessment tool prepared for the BTO 101 - Information Technology in 
Education-I course offered at the Department of Computer Education and Instructional Technologies at Anadolu University. 
The software has a reader and an author version. It has the ability to diagnose item quality through item difficulty, item 
discrimination and distractor efficiency indices. However, new qualifications will be added to the current software for further 
developmental stages. The software will be able to: 

categorize questions as easy, moderate and difficult, and list questions from easiest to the most difficult, 
calculate test reliability and generate reliable test booklets on specific subject matters, 
prepare question sets in accordance with the degree of difficulty desired by instructors. 

The online assessment environment presented in the present paper probably poses several limitations since it was developed 
within a short span of time by a small design team. Therefore, it can be considered a demo for evaluative purposes. Formative 
evaluation of the program by students, teachers and instructional designers is necessary, so that, the instrument can serve 
better each time it is used. Our next step is to delve into opinions of participants who used the formative assessment tool for 
their course exercises. 
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